refactor(file): composable Lines iterator; deprecate channel-based ReadFile helpers#742
Merged
Mzack9999 merged 2 commits intoprojectdiscovery:mainfrom May 8, 2026
Merged
Conversation
…very#719) When a tool reads a list of values from a file (resolvers, wordlists, etc.) it's common for users to mix one-per-line and comma-separated forms on the same line. The two new helpers stream non-empty values from the file/reader, splitting each scanned line on the supplied runes and trimming whitespace; passing no separators reduces to the existing ReadFile/ReadFileWithReader behaviour with TrimSpace applied. Closes projectdiscovery#719
Neo - PR Security ReviewNo security issues found Comment |
Mzack9999
approved these changes
May 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #719.
Supersedes the original
ReadFileSplit/ReadFileWithReaderSplitproposal. Per review discussion: rather than add yet anotherReadFile*variant (each new option adds a new combinatorial helper), introduce a single composable iterator and reduce the existing helpers to thin deprecated wrappers over it.What
New file
file/lines.goexposes one primitive per source, returning a Go 1.23+ iterator:Composable options (applied in order
scan → split → trim → skip-empty → filter):The resolver-file use case from #719 / httpx#2351 becomes:
Why this shape
ReadFile,ReadFileWithBufferSize,ReadFileSplit,ReadFileSplitWithBufferSize,…WithReaderAnd…, etc. One iterator plus N orthogonal options replaces all of them and any future variants (WithFilter,WithComment, …) compose for free.os.Openfailures inside the goroutine andscanner.Err()entirely.iter.Seq2[string, error]makes both visible: the iterator yields a final("", err)pair and stops.iter.Seq2cancels naturally onbreak; the file is closed viadeferwhen iteration ends. The old channel pattern leaked a goroutine forever if the consumer stopped reading early.strings.Linesprecedent.rangestep.Deprecation
The four existing exported helpers keep their
(chan string, error)signatures (zero source-level breakage) and are marked// Deprecated:with the migration recipe. Their bodies now pump from the new iterator into the channel, so behaviour is byte-for-byte identical and the existingTestReadFile*tests pass unmodified.ReadFile(path)Lines(path)ReadFileWithBufferSize(path, n)Lines(path, WithBufferSize(n))ReadFileWithReader(r)LinesReader(r)ReadFileWithReaderAndBufferSize(r, n)LinesReader(r, WithBufferSize(n))The
ReadFileSplit/ReadFileWithReaderSplit/splitLineByRuneshelpers from the earlier revision of this PR are removed; their functionality is now expressed asLines(path, WithSplit(','), WithTrimSpace(), WithSkipEmpty()).Tests
file/lines_test.go(14 tests): each option in isolation, the resolver-file end-to-end scenario, missing-file error path, early-breakbehaviour, scanner-error propagation throughLinesReader, and reader-side equivalents.file/file_split_test.gowas removed along with the helpers it covered.TestReadFile,TestReadFileWithBufferSize,TestReadFileWithReader,TestReadFileWithReaderAndBufferSizeare unchanged and still pass — they validate that the deprecated wrappers preserve exact prior behaviour (including emitting empty lines and not trimming).Notes for downstream tools
After this lands,
httpx/nuclei/subfinderetc. can migrate at their own pace:for line := range fileutil.ReadFile(path)→for line, err := range fileutil.Lines(path).Lines(path, WithSplit(','), WithTrimSpace(), WithSkipEmpty()).No flag-day required; the deprecated channel functions stay until consumers have moved.